Papers with next-word prediction
Bias in Language Models: Beyond Trick Tests and Towards RUTEd Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Standard bias benchmarks are used for large language models to measure the association between social attributes and single-word outputs. |
| Approach: | They adapt three standard bias metrics of next-word prediction to measure gender-occupation bias and develop an analogous RUTEd evaluation in three contexts of real-world LLM use. |
| Outcome: | The proposed benchmarks are robust to lengthening model outputs via a more realistic user prompt in the domain of gender-occupation bias. |
Great Memory, Shallow Reasoning: Limits of kNN-LMs (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing models trained on poor quality data have shown strong performance in language modeling and some downstream benchmarks. |
| Approach: | They evaluate kNN-LMs on a diverse set of tasks and evaluate their performance. |
| Outcome: | The proposed extension could improve on a variety of tasks, but it fails to perform on reasoning tasks that require integrating multiple pieces of information. |
ToW: Thoughts of Words Improve Reasoning in Large Language Models (2025.naacl-long)
Copied to clipboard
Zhikun Xu, Ming Shen, Jacob Dineen, Zhaonan Li, Xiao Ye, Shijie Lu, Aswin Rrv, Chitta Baral, Ben Zhou
| Challenge: | Unlike other data augmentation methods, thoughts of words (TOW) views next-word prediction as a core reasoning task and injects fine-grained thoughts into pre-training texts. |
| Approach: | They propose a training-time data-augmentation method called thoughts of words (TOW) that injects fine-grained thoughts directly into a next-word prediction task and teaches the model to understand how the observed next word is related to previous contexts. |
| Outcome: | The proposed method reduces model hallucination by 10% and improves reasoning performance by 7% to 9% on average. |
Improving Adversarial Text Generation by Modeling the Distant Future (2020.acl-main)
Copied to clipboard
Ruiyi Zhang, Changyou Chen, Zhe Gan, Wenlin Wang, Dinghan Shen, Guoyin Wang, Zheng Wen, Lawrence Carin
| Challenge: | Recent work has shown excellent performance on text generation tasks by combining reinforcement learning (RL) and generative models. |
| Approach: | They propose a model-based imitation-learning approach to improve text generation performance by focusing on a long horizon. |
| Outcome: | The proposed model improves on a number of text-generation tasks and provides intermediate rewards for generator optimization. |
Phased Instruction Fine-Tuning for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to enhance pre-trained language models' ability to follow instructions are limited due to the simultaneous handling of varying instruction complexities. |
| Approach: | They propose a phased instruction fine-tuning method that posits that the transition of a pre-trained language model from simple next-word prediction to sophisticated instruction following is a gradual learning process. |
| Outcome: | The proposed method surpasses the one-off instruction fine-tuning method in win rate and validates the hypothesis of progressive alignment. |
How Do Neural Sequence Models Generalize? Local and Global Cues for Out-of-Distribution Prediction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Using RNN and transformer language models, we show consistent generalization in out-of-distribution contexts. |
| Approach: | They propose two idealized models of generalization in next-word prediction . they show that neural language models interpolate between these two forms of generalisation . |
| Outcome: | The proposed models exhibit consistent generalization in out-of-distribution contexts. |
Like a Baby: Visually Situated Neural Language Acquisition (P19-1)
Copied to clipboard
| Challenge: | A multi-modal neural architecture outperforms its equivalent trained on language alone with a 2% decrease in perplexity . |
| Approach: | They propose to use visual context to train neural language models to perform next-word prediction. |
| Outcome: | The proposed model outperforms its equivalent trained on language with 2% decrease in perplexity even when no visual context is available at test. |
ClozeMath: Improving Mathematical Reasoning in Language Models by Learning to Fill Equations (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train large language models do not capture how humans learn to think. |
| Approach: | They propose a method to fine-tune large language models for mathematical reasoning by using a text-infilling task that predicts masked equations from a given solution. |
| Outcome: | Experiments on GSM8K, MATH, and GSM-Symbolic show that ClozeMath surpasses baseline Masked Thought in performance and robustness with two test-time scaling decoding algorithms, Beam Search and Chain-of-Thought decoding. |
Language models and brains align due to more than next-word prediction and word-level information (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models have been shown to significantly predict brain recordings of people comprehending language. |
| Approach: | They propose to use two perturbations to design contrasts that control for different types of information. |
| Outcome: | The proposed model is largely agnostic about the exact linguistic information contained in the conceptual quantities "word-level information" and "multi-word information". |
JoPA: Explaining Large Language Model’s Generation via Joint Prompt Attribution (2025.acl-long)
Copied to clipboard
| Challenge: | Existing attempts to explain the entire language generation often treat input prompt texts independently, ignoring their combinatorial effects on the follow-up generation. |
| Approach: | They propose a framework for explaining how a few prompt texts collaboratively influences the LLM's complete generation. |
| Outcome: | The proposed explanations demonstrate faithfulness and efficiency of the proposed framework. |
Resource-Rational Noisy-Channel Language Processing: Testing the Effect of Algorithmic Constraints on Inferences (2025.emnlp-main)
Copied to clipboard
| Challenge: | a fundamental question in psycholinguistics is how comprehenders form interpretations of utterances that they hear or see. |
| Approach: | They propose to use a language model as a prior and an error model to encode likelihoods to perform incremental and approximate probabilistic inferences over intended sentences and production errors. |
| Outcome: | The proposed model captures previously established patterns in human sentence processing, and trade-off between human-like noisy-channel inferences and computational resources falls out of the model. |
From Language to Cognition: How LLMs Outgrow the Human Language Network (2025.emnlp-main)
Copied to clipboard
Badr AlKhamissi, Greta Tuckute, Yingtian Tang, Taha Osama A Binhuraib, Antoine Bosselut, Martin Schrimpf
| Challenge: | Large language models exhibit remarkable similarity to neural activity in the human language network, but their properties remain unclear. |
| Approach: | They benchmark 34 training checkpoints spanning 300B tokens across 8 different model sizes . they find that brain alignment tracks the development of formal linguistic competence more closely than functional linguistic competency. |
| Outcome: | The results show that large language models exhibit similarity to human language networks . they show that the correlation between next-word prediction and brain alignment fades once models surpass human language proficiency. |
Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-training (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for enhancing cross-lingual transfer are limited by parallel resources and lack linguistic and domain coverage. |
| Approach: | They propose a cross-lingual in-context pre-training approach that leverages semantically related bilingual Wikipedia documents to enhance cross-linguistic transfer. |
| Outcome: | The proposed approach improves multilingual performance on three models across six target languages. |
An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisal (2026.acl-long)
Copied to clipboard
| Challenge: | Surprisal theory claims that difficulty of sentences increases linearly with surprise . a neural LM that can explain garden-path effects cannot be built, says a new study . |
| Approach: | They propose to fine-tune neural LMs to better align surprisal-based reading-time estimates with actual reading times. |
| Outcome: | a new study shows that fine-tuned neural LMs do not overfit on held-out items . the results show that they improve predictive power for human reading times . |